Add line-caption generator for raw portrait shorts by natashaannn · Pull Request #91 · ragTechDev/deckcreate

natashaannn · 2026-07-06T15:40:02Z

Summary

Adds a standalone pipeline that takes a raw short-form portrait video and produces a burned-in-caption render, independent of the existing longform/shorts camera+hook pipeline:

npm run captions:create -- --video <path> [--num-speakers N] — copies the source video, extracts audio, transcribes (Transcriber), force-aligns for exact word boundaries, diarizes + assigns speakers when --num-speakers > 1, chunks the transcript into fixed 3-word CaptionLines (merging whisper's BPE sub-word tokens back into whole words so a line never splits mid-word, and never spanning a segment/speaker boundary so a line never mixes two speakers), and writes public/line-captions/{id}/lines.json + a human-editable lines.doc.txt.
npm run captions:merge -- --id <slug> — reparses lines.doc.txt after a human reword pass and overwrites only the text field per [id]; startMs/endMs/speaker stay untouched (best-effort timing, no re-alignment).
npm run captions:render -- --id <slug> — renders the new LineCaptionClip-{id} Remotion composition (plain video pass-through, no jump cuts/camera profiles, plus the new LineCaptionOverlay) to public/renders/.

Reuses existing primitives unmodified: Transcriber, align-transcript.js, diarize-audio.js/assign-speakers.js, the SHORT_IDS per-clip Composition registration pattern in Root.tsx, and stampMetadata.

Also includes two unrelated pre-existing fixes discovered while getting this branch to push cleanly:

scripts/config/paths.test.ts hardcoded POSIX path separators, failing all 26 of its assertions on Windows. Rewritten to build expectations with path.join (test-only change; paths.ts itself was already correct).
.husky/pre-push's secret-token scan wasn't excluding .venv/, so a base64 font-glyph blob in a vendored PIL file false-positived as a leaked API key and blocked every push on machines with a local .venv. Added --exclude-dir=.venv alongside the existing node_modules/.next exclusions.

Full design rationale and the step-by-step build log live in docs/implementation-guides/LINE_CAPTION_SHORTS.md. New lines.json schema and source files are documented in CLAUDE.md.

Test plan

npm run test:unit — full suite passes (280 passed, 2 pre-existing skips, 0 failed), including new unit tests: chunkLines.test.js (BPE-word reconstruction, punctuation attach, speaker-boundary breaking, cut exclusion, t_end fallback), create-line-captions.test.js (buildLineDoc formatting), merge-line-captions.test.js (doc parsing + text-only merge), LineCaptionOverlay.test.tsx (active-line selection, speaker tinting).
tsc --noEmit — passes.
npm run lint — passes on all new/changed files.
Pre-push hook (full jest suite, debugger/.only()/secret scans, npm audit, runtime-dir gitignore check) — passes clean.
Manual, not yet run (needs a real sample clip + this repo's WhisperX/Docker alignment setup, unavailable in the sandbox this branch was built in):
1. npm run captions:create -- --video <sample.mp4> --num-speakers 2 on a short real two-speaker portrait clip.
2. Open public/line-captions/{id}/lines.doc.txt — confirm 3-word lines read naturally, grouped under === SPEAKER === headers, and no line mixes two speakers.
3. Hand-edit one line's wording.
4. npm run captions:merge -- --id {id}, confirm only that line's text changed in lines.json and its startMs/endMs are untouched.
5. npx remotion studio, select LineCaptionClip-{id}, confirm captions are burned in, timed correctly against the audio, and the edited line shows its new wording during its original time window.
6. npm run captions:render -- --id {id}, confirm public/renders/{id}.mp4 is produced.

🤖 Generated with Claude Code

Co-Authored-By: Claude Sonnet 5 <[email protected]>

Reconstructs whisper's BPE-level tokens into whole words before bucketing into fixed-size caption lines, so a line never splits a word or counts punctuation as its own slot. Lines never span a segment boundary, which keeps two speakers from ever sharing a line. Co-Authored-By: Claude Sonnet 5 <[email protected]>

Orchestrates the raw-portrait-video-to-caption-lines pipeline: copy source, extract audio, transcribe, force-align for exact word boundaries, diarize and assign speakers when --num-speakers > 1, then chunk into 3-word lines and write lines.json + a human-editable lines.doc.txt. Reuses the existing Transcriber class and align/diarize CLI scripts unmodified. Co-Authored-By: Claude Sonnet 5 <[email protected]>

Re-parses lines.doc.txt after a human reword pass and overwrites only the text field of each matching lines.json entry by [id] — startMs/endMs/speaker are left untouched, so the caption keeps its original best-effort time window regardless of how much the wording changed. Co-Authored-By: Claude Sonnet 5 <[email protected]>

A plain video pass-through (no jump cuts, no camera profiles) plus LineCaptionOverlay, which shows fixed-window 3-word caption lines instead of the timing-driven grouping CaptionOverlay uses. Registers one LineCaptionClip-{id} composition per public/line-captions/*/lines.json, mirroring the existing ShortFormClip-{id} pattern in Root.tsx. Co-Authored-By: Claude Sonnet 5 <[email protected]>

Modeled on scripts/shorts/render-short.js — spawns remotion render against the LineCaptionClip-{id} composition and writes to public/renders/. Co-Authored-By: Claude Sonnet 5 <[email protected]>

Co-Authored-By: Claude Sonnet 5 <[email protected]>

Expected strings were hardcoded with POSIX forward slashes, so every test failed on Windows where path.join produces backslashes. Build expectations with path.join instead of literal strings — the production path.ts code was already correct, only the test's assertions were platform-specific. Co-Authored-By: Claude Sonnet 5 <[email protected]>

A vendored, gitignored Python virtualenv (.venv/Lib/site-packages/PIL/ ImageFont.py) contains a base64 font-glyph blob that coincidentally matches the sk-[A-Za-z0-9]{48} pattern, false-positiving as a leaked key and blocking every push on machines with a local .venv present. Exclude it the same way node_modules and .next already are. Co-Authored-By: Claude Sonnet 5 <[email protected]>

ytexplorer

Manual end-to-end test on Windows — 3 render fixes

Nice work on this pipeline. I pulled the branch and ran the unchecked manual test-plan box end-to-end on Windows 11 (Node 24, Python 3.12 venv, whisper.cpp medium.en, whisperx, diarize) against a real 52 s two-speaker portrait clip.

Works great: captions:create (transcribe → align → diarize → chunk) is solid — diarization found 2 real speakers, lines.doc.txt is correctly chunked with no speaker-mixing, and captions:merge changed only the edited line's text across all 73 lines (timings/speaker preserved).

Three issues blocked captions:render; all three are fixed on my branch ytexplorer:feature/line-captions (full jest suite + lint + audit pass), and I've left inline suggestions below for the two that live in this file:

--props inline JSON breaks on Windows — with spawn(shell:true), cmd.exe strips the quotes and Remotion gets unparseable JSON, aborting before bundling. → props via temp file (inline suggestion).
--outName isn't honored by remotion render — combined with Config.setOutputLocation('public/renders') the file lands at public/renders.mp4 instead of public/renders/<id>.mp4. → positional output path + ensureDir (inline suggestion).
Remotion version mismatch (lockfile) — remotion/@remotion/gif/captions/sfx/install-whisper-cpp/eslint-plugin are pinned at 4.0.451 while @remotion/cli/bundler/renderer/media-parser resolve to 4.0.477. Remotion requires the whole family on a single version, so this split lockfile fails LineCaptionClip renders (the core remotion at 4.0.451 vs the 4.0.477 tooling is the main gap). Can't suggest inline (it's the lockfile), but it's ready to cherry-pick: ytexplorer/deckcreate@6b9da81 — realigns the whole family to 4.0.477.

ytexplorer · 2026-07-07T16:41:11Z

+ *   node scripts/line-captions/render-line-captions.js --id <slug>
+ */
+
+import fs from 'fs-extra';


Need os for the temp props file below.

Suggested change

import fs from 'fs-extra';

import fs from 'fs-extra';

import os from 'os';

ytexplorer · 2026-07-07T16:41:11Z

+  const outName = `${args.id}.mp4`;
+  console.log(`Rendering line-caption clip: ${args.id}`);
+  console.log(`Output file: ${outName}`);
+
+  const renderArgs = [
+    'remotion', 'render',
+    'remotion/index.ts',
+    `LineCaptionClip-${args.id}`,
+    '--outName', outName,
+    '--props', JSON.stringify({
+      linesSrc: `line-captions/${args.id}/lines.json`,
+      brandSrc: 'brand.json',
+    }),
+  ];


Write props to a temp file (Windows quote-safe) and pass the full output path positionally so it lands in public/renders/<id>.mp4.

Suggested change

const outName = `${args.id}.mp4`;

console.log(`Rendering line-caption clip: ${args.id}`);

console.log(`Output file: ${outName}`);

const renderArgs = [

'remotion', 'render',

'remotion/index.ts',

`LineCaptionClip-${args.id}`,

'--outName', outName,

'--props', JSON.stringify({

linesSrc: `line-captions/${args.id}/lines.json`,

brandSrc: 'brand.json',

}),

];

const outName = `${args.id}.mp4`;

console.log(`Rendering line-caption clip: ${args.id}`);

console.log(`Output file: ${outName}`);

// Ensure the output directory exists so the positional output path below has

// a parent to write into on a fresh checkout.

await fs.ensureDir(path.join(cwd, 'public', 'renders'));

// Pass props via a temp file rather than inline. Inline `--props '<json>'`

// has its double quotes stripped by cmd.exe (spawn shell:true on win32), so

// Remotion receives unparseable JSON and aborts before bundling.

const propsFile = path.join(os.tmpdir(), `line-caption-props-${args.id}-${Date.now()}.json`);

await fs.writeJson(propsFile, {

linesSrc: `line-captions/${args.id}/lines.json`,

brandSrc: 'brand.json',

});

const renderArgs = [

'remotion', 'render',

'remotion/index.ts',

`LineCaptionClip-${args.id}`,

// Full output path positionally. `--outName` is not honored by

// `remotion render`; with Config.setOutputLocation('public/renders') it

// writes public/renders.mp4 instead of public/renders/<id>.mp4.

`public/renders/${outName}`,

`--props=${propsFile}`,

];

ytexplorer · 2026-07-07T16:41:11Z

+  await new Promise((resolve, reject) => {
+    const proc = spawn('npx', renderArgs, {
+      stdio: 'inherit',
+      cwd,
+      shell: process.platform === 'win32',
+    });
+    proc.on('close', code => code === 0 ? resolve() : reject(new Error(`Render exited ${code}`)));
+    proc.on('error', e => reject(e));
+  });


Wrap the render in try/finally so the temp props file is cleaned up.

Suggested change

await new Promise((resolve, reject) => {

const proc = spawn('npx', renderArgs, {

stdio: 'inherit',

cwd,

shell: process.platform === 'win32',

});

proc.on('close', code => code === 0 ? resolve() : reject(new Error(`Render exited ${code}`)));

proc.on('error', e => reject(e));

});

try {

await new Promise((resolve, reject) => {

const proc = spawn('npx', renderArgs, {

stdio: 'inherit',

cwd,

shell: process.platform === 'win32',

});

proc.on('close', code => code === 0 ? resolve() : reject(new Error(`Render exited ${code}`)));

proc.on('error', e => reject(e));

});

} finally {

await fs.remove(propsFile).catch(() => {});

}

natashaannn and others added 9 commits July 6, 2026 18:29

docs: add implementation guide for line-caption shorts

6b2179c

Co-Authored-By: Claude Sonnet 5 <[email protected]>

feat: add captions:render script and npm scripts

b9ed120

Modeled on scripts/shorts/render-short.js — spawns remotion render against the LineCaptionClip-{id} composition and writes to public/renders/. Co-Authored-By: Claude Sonnet 5 <[email protected]>

docs: document lines.json schema and line-caption source files

5ee504b

Co-Authored-By: Claude Sonnet 5 <[email protected]>

ytexplorer reviewed Jul 7, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add line-caption generator for raw portrait shorts#91

Add line-caption generator for raw portrait shorts#91
natashaannn wants to merge 9 commits into
mainfrom
feature/line-captions

natashaannn commented Jul 6, 2026

Uh oh!

ytexplorer left a comment •

edited

Loading

Uh oh!

ytexplorer Jul 7, 2026

Uh oh!

ytexplorer Jul 7, 2026

Uh oh!

ytexplorer Jul 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	import fs from 'fs-extra';
	import fs from 'fs-extra';
	import os from 'os';

Uh oh!

Conversation

natashaannn commented Jul 6, 2026

Summary

Test plan

Uh oh!

ytexplorer left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ytexplorer Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

ytexplorer Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

ytexplorer Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ytexplorer left a comment •

edited

Loading